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Abstract 

We present generalized delayed neural network (DNN) model with positive delay feedback and 
neuron history. The local stability analysis around trivial local equilibria of delayed neural 
networks has applied and determine the conditions for the existence of zero root. We develop 
few innovative delayed neural network models in different dimensions through transformation 
and extension of some existing models. We found that zero root can have multiplicty two under 
certain conditions. We further show how the characteristic equation can have zero root and 
its multiplicity is dependent on the conditions undertaken. Finally, we generalize the neural 
network of N neurons through which we determine the general form of Jacobian of the linear 
form and corresponding characteristic equation of the system. 


Introduction 

The stability analysis in various dynamical systems allows us to understand near equilibrium 
properties of flows of the systems under pertubations and stability criteria around the equilibria. 
An equilbrium is locally stable if a system near the equilibrium approaches towards it [T], and 
will exhibit critical behavior of the system near that equilibrium point but may be different for 
different local equilibrium points. The local stability analysis, in general refers to the procedure 
of approximating a non-linear model by a linear one in the vicinity of local equilibrium point and 
analyzing the behavior of the linear form of the system. The local stability of the system can be 
characterized and analyzed by determining the eigenvalues of the linear system calculated from 
the so-called Jacobian matrix of the system [T]. 

The stability analysis have been applied to various dynamical systems such as exponential 
and logistic models of population growth, models of natural selection, delayed models of neural 
netwroks [T] etc. The dynamics of natural systems (physical, chemical, biological systems) 
involves with delay time in the process, and can be modeled by delayed differential equations 
(DDEs) which allows to understand the important role of delay time, and have been an extensive 
area of research since the past few decades in a wide range of applications. The most fundamental 
functional differential equation (FDE) is the linear hrst order DDE which is of the form [2] 


dx 

dt 


/i(t)x(t)-F/ 2 (t)x(t - r), t>0 


( 1 ) 
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where, /i and /2 are time dependent functions, r is the time delay involved in the time evolution 
of vector x. Such mathematical modeling is very useful in describing neuron interactions with 
different time delays which are generally reflected in experimental data such as EEG, EMG 
data and fMRI, DTI data etc. In such situation, the time delays involved in DDEs have been 
incorporated in the mathematical models of delayed neural networks which represent the prop¬ 
agation time among the axons and the dendrites. Eor coupled system of DNNs, the dynamics 
involves the excitatory and inhibitory influences and the activation level of one neuron is affected 
by the local feedback from the other neurons [3]. However, to gain a better insight into such 
non-equilibrium dynamical systems, one needs to focus on the equilibria and its stability. 

The delayed systems of differential equations modeling of neural networks have been a 
cutting-edge research in the area of neuroscience. Both single variable and multivariate dis¬ 
crete and continuous time models for DNNs have been developed and investigated in different 
dimensions. For instance, Liao et al. [3] studied a single delayed neuron model, while elaborate 
discussions of two-neuron model was discussed in different works [HHE]. Further, tri-neurons 
networks have also been studied with different feedback models [2l[3l[8HT^ and four-neuron 
BAM |29H31j models are discussed upto some extent. However, there has not been a generalised 
model of neural network from which a simpler model can be deduced, and the general Jacobian 
and the corresponding characteristic equation are still left as open questions. We focus on gen¬ 
eralization of the models to N-dimensions to present the outsets of these innovative models and 
demonstrate the corresponding generalisations of the underlying properties. The perspective for 
the extension or generalisation is, however, referred to some of the existing models. 


In this paper, we focus on the linear stability analysis of DNNs by linearising the model 
equations around the trivial local equilibria. Initially, the typical non-linear function ‘tanh’ 
is associated with the DDEs of the models which is then expanded using the Taylor series 


expansion and the higher order terms are truncatec 


linear terms. We consider the Jacobian, 


d{xx,...,Xn) 


a(Ci,...,c'„) 


during the linearisation leaving only the 
= 0, where n is the number of neurons, 


to obtain the characteristic equation of the system and determine its zero root. This brief 
delineates the conditions for the subsistence of multiplicity 2 of the zero root, as this leads to 
the Bogdanov-Takens bifurcation PE]. One can use the centre manifold reduction [19II24] and 
the normal form method [251128] to compute the simpler normal form of the original DDEs and 
analyse the dynamic behaviors of the system DNNs p. Furthermore, we observe similarity 
in the Jacobians and the corresponding characteristic equations of the systems and from this 
standpoint, we generalize the form of the Jacobian and the corresponding characteristic equation 
to N dimensions. 


Local Stability Analysis 

The local or linear stability analysis of DDE can be done by taking into account the linear 
form of the non-linear DDEs around any local equilibrium ([T|). This linearisation is achieved by 
Taylor expansion of the non-linear function involved in DDEs. To deal with such equations for 
the non-trivial solutions, one assumes that the solution exists in the linear form x = , C 

G C (complex space) and the model DDEs will reduce to characteristic equation of the system 
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© wm- We consider the DDEs which are constant-coefficient equations of the following form, 

^ = fix{t) + f 2 x{t - t), t>0 (2) 

The following the linearization process of the above form of equations, one can find the the 
following characteristic equation: 


(A - /i) - /2e-^" = 0 (3) 

Analyzing the above equation ([3|), we check for the existence of the zero root and the necessary 
conditions for the subsistence of its multiplicity. 

ID Dynamic Neural Network model 

The one dimensional neural network (NN) which we consider here is slightly different from that 
investigated by Liao et al. in [3] in the arguments of the non-linear function in the model. The 
model is the simplication of the model considered in [5] which is discussed briefly in the above 
section. The model equation is given by 

dx 

— = —x{t) + a tanh(x(f — r/)) (4) 

where a and Tf have the usual meanings. We study the linear stability analysis in the vicinity 
of the trivial equilibrium of the system ([3]), assuming that it always possesses an equilibrium 
at the origin and the solution exists in the linear form of x = where A is the eigenvalue 

and C £ C. Using Taylor series approximation on the sigmoid function ’tanh’ around the trivial 
equilibrium reduces to the linear form ^ = —x{t) + ax{t — Tf). The corresponding characteristic 
equation can be obtained as, 

= (A + l)-ae“^^^ =0 (5) 

where, is the functional form of order 0. 

Lemma 1.1: If xq is a root of F^^\x) of multiplicity r, then F^^\x) also has xq as its root of 
multiplicity r — 1 [7]. In other words, a root of F^^\x) having a multiplicity r is a single root of 
F^^\x) and not a root of (x). 

Lemma 1.2: A = 0 is a single root oi ^ iff a = 1. 

Proof Suppose A = 0 is a single root of ([5]). Then, 

F(0) = l-a = 0 (6) 

which gives a = 1. For the other direction, by Lemma 1.1, we need to claim F^^\o) = 0 and 
/ 0 given that a = 1. Clearly, = 0 and = —Tf. 



4 


2D Dynamic Neural Network model 

We consider the Delayed NN model considered by Fan et al. in [S] which is different from the 
dynamic behaviors of two-neuron networks studied in |14H18j . The network is being modeled, 
along with the introduction of delayed self-feedback and a delayed connection from the other 
neuron, by a coupled system of DDEs as follows: 

—xi{t) + a tanh(3:i(t — rj)) — ai2tanh(3:2(t — T 2 )), 

—X 2 {t) — a tanh(3:2(t — tj)) -|- Q;2itanh(3:i(t — ri)) (7) 

where 012 and 021 are the connection strengths with r* (z = 1, 2) as the connection delays and 
a > 0 is the feedback strength having a delay rj. The linearized form of d?]) is given by, 

-xi{t) + axi{t - Tf) - ai 2 X 2 it - T 2 ), 

-X2{t) - aX2{t - Tf) + a2lXl{t - Tl) (8) 

The corresponding characteristic equation calculated from the Jacobian of equation Q in A is 
given by, 

F(A) = (1 + A)2 - + 0120216 ^^^ = 0 (9) 

where, r = 

Theorem 2.1: (z) The characteristic equation Q has a zero root (A 

iff 0 

1 2 T + l +1 

r > Tf, and a = -, 0120:21 =- 

T-Tf T-Tf 

(a) The maximum multiplicity of the zero root is 2. 

Proof (z) First suppose that A = 0 is a root of ([9|) of multiplicity 2. This results into the 
equations: 

F(0) = 0 and F(^^(0) = 0 

^ 1 — -|- 012021 = 0 and 1 -|- o^r/ — oi 202 ir = 0 

Solving these equations yields the conditions (fTO|) . 

For the other direction, by Lemma nl, it would be suffice to show = 0, =0 

and / 0 successively when o^ and 012 O 21 satisfy the conditions (fTUl) . Eventually, (0) 

= 0, =0 and = 1 -|- 2 (t -|- Tf). 

(zz) We take into account the method of contradiction to prove this statement. Suppose A 
= 0 is a multiple root of ([9]) having multiplicity 3. This necessarily implies F"'(0) = 0 (n = 0, 
1, 2), where n denotes the number of derivatives w.r.t. A. Substituting the values of c? and 
0 : 120:21 in F(2)(0) = 0, we obtain t + Tf = - However, t > Tf and r > 0. This contradicts 
our supposition and hence the proof is complete. 


= 0) of multiplicity 2 
( 10 ) 


dxi 

dt 

dx2 

dt 


dxi 

dt 

dX2 

dt 
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3D Dynamic Neural Network model 


In three neurons model, each neuron has the ability to activate itself and the activation is 
dependent on the history of its previous activation [3]. The axonal and dendritic propagation 
time, also called the synaptic delay, is considered associated with the local positive feedback, 
biologically termed ‘reverberation’ [ 6 ]. The model is described by, 

dx' 

-^ =-Xi{t) + aii?mh[xi +2 - lixi+ 2 {t - t)], r>0 (11) 

where i = 1,2, 3, Xj represents the activation level of the neuron with the activity coefficient 
aj, /3 denotes the inhibitory influence measure of the past history, Xi +2 is the reverberation. The 
linearization of equation (|lll) allows to obtain the following equation, 


Xi{t) = -Xi{t) + ai[xi+2 - Pxi+2{t - r)] 


( 12 ) 


and the corresponding characteristic equation at the equilibrium point (origin) can be obtained 
as below. 


A + 1 0 

-02(1-/36-^") A + 1 

0 -03(1 - / 3 e-^^) 


—ai(l — /3e '^’') 
0 

A + 1 


= 0 


which reduces to the following functional form, 

F(A) = (A + 1)^ - 0123(1 - = 0; aijk... = a^ajak--.. 


(13) 


Thus, the characteristic equation (1131) contains the terms (A + 1) and (1 — with degree 3. 

If /3 = 1 - ^ and 1 < a < 4, where a = then (fT3]) has a zero root of multiplicity two and 

no purely imaginary roots. The conditions of single root and double root are briefly discussed 

in p]. 


4D Dynamic Neural Network model 


We then present an innovative approach to the delayed networks containing four neurons, by 
extending the 3D DNN model. The features and the forms of equations are equivalent, except 
for the introduction of cross-linking among the neurons. As a result, there arises different 
measures of the inhibitory influence of the past history as shown in Fig. ©• The delayed 
bidirectional associative memory(BAM) neural network of four neurons with time delays have 
been investigated extensively in [29H3T] . 

The different measures signifies that the inhibitory influences vary with different time delays. 
Moreover, the local positive feedback is also increased. The model is given by. 


dxi 

dt 


-Xi{t) + a+anh Xi+ 2 (t) - /3iXi+2(i - n) 


+ ajtanh 


Xi+3(t) - l32Xi+3(t - T 2 ) 


(14) 


where i = 1,2,3,4, /3i and (32 are the measures of inhibitory influences during time delays ti 
and T 2 respectively, Xj +2 and Xj +3 are the feedback, and Xi and a* have the usual meanings. 
The linearization of (I14p and keeping t\ = T 2 = t, we get the following linearized form. 


dxi 

dt 


-Xi(t) + ai Xi+2{t) - /3iXi+2it - t) 


a,; 


Xi+^{t) - l32Xi+^{t - t) +0{t^) 


(15) 
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+ excitatory 
- inhibitory 


-PjA-iCt-T) 



Figure 1. Architecture of the four-neuron model 


The corresponding Jacobian matrix of the model is given by, 


X + 1 0 -ai/i - 0 : 1/2 

—02/2 A -|- 1 0 — 02/1 

—03/1 —03/2 A -|-1 0 

0 -04/1 -04/2 A -h 1 


where, /j(t) = Expanding the determinant, we arrive at the following characteristic 

equation, 

F(A) = {X + 1 )'^ - 01234/1 + giX, t) 


where, the function g{X, r) is given by, 

5 (A,t) = (A -|- l)^/f (oi 3 — O24) — (A -I- l)/l/ 2 [oi 24 +/ 2 (oi 23 + O134)] 

+ [01234/2 “ 0234/1/2] (16) 


Theorem 4.1: Suppose /3i = 1 and /32 = 1 + €4 3 , then 


(i) A = 0 is a zero root of multiplicity 1 iff t ^ ^^ - and 

3e4 ^+3+62^4 ^—£4 + 

(a) A = 0 is a zero root of multiplicity 2 iff t = —i-^4--r 

3e4^-|-3-|-e2e4 £4 ^(es+es) 

where ei = 013 — 024 , £2 = 0124 , £3 = 0123 + 0134 , €4 = 01234 and €5 = 0234 - 
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Proof We claim the above theorem using Lemma 1.1. Clearly, 

F{0) = 1 + (1 - /3i)2ei - (1 - /3i)(l - P2)[e2 + (1 - ^ 2 ) 63 ] + [^4(1 - A)" + £ 4(1 - 132? - £ 5(1 - /3i)(l - (32?] 
Substituting the aforementioned values of /3i and (32 ultimately yields F(0) =0. And, 

Fd)(A) = 4(A + 1)3 + (A + 1)2(1 - die-^")(2di£ir) + (A + 1) f2ei(l - - (-2rdi/32e-2^" + r(di 


+ p2)e-n?2 + (1 - /32e-^")£3) - (/?i/32e-2^" - (/3i + /32)e-^" + l)(/32re3e-^") - {^ 1 ^ 20 -^^^ - (/?i + 


/ 32 )e + l)(e2 + (1 -/ 32 e ^^€3 + 2 e 5 / 32 T) - 4 e 4 / 3 iT(l -/ 32 e ^^)3 - (1 -/ 32 e ^^) 2 ( 3 e 4 / 32 T + esdi^e ^'^) 


( 0 ) — 4 + T — e2£4 ^ + e4 ^ (gg — 3e4(l + £4 ^) + £5) 


> 0 


If r = 


4 

—I-n-r"r--- 

3e4^H-3+e2e4 ^—^4 


, it ultimately follows that r — £264 3+64 s (63 — 3 e 4 (l + 


^) + ^ 5 ) = —4 and hence, F(^)(0) = 0, F(^?0) / 0. This completes the proofs. 


5D Dynamic Neural Network model 


We now present the 5D DNN model by extending the 4D DNN model. The increase of one 
neuron, however, adds up another cyclic pathway of neurons in a loop. As a result, there are 
increased influences and feedback from other neurons and their past history. The model DDEs 
are given by, 


dxi 

dt 


-Xi(t) + Oitanh Xi+2(t) - /3iXi+2(t - n) 


+ ajtanh 


Xi+S?) - (32Xi+'i{t 



+Q;jtanh 


Xi+i{t) - l3i,Xi+i{t - T 3 ) 


(17) 


where i = 1,2,3,4, 5, (dj measures the inhibitory influences during time delays Tj respectively 
where j = {1, 2, 3}; Xi +4 are the feedback, and other notations have the usual meanings. The 
linear form of (|17l) is given by. 


Xi{t) 


-Xi{t) + Ui 


Xi+2{t) - ?lXi+2{t 



+ Oli 


Xi+sft) - l32Xi+3(t - T 2 ) 


-\-OLi 


Xi+i{t) - / 33 Xi+ 4 (t - T 3 ) 


(18) 


Assuming the delays are identical and the solutions exist in the linear form as in the previous 
models, the corresponding Jacobian is given by, 


A + 1 

0 

-ai/i 

-ai /2 

-ai /3 

-“ 2/3 

A + 1 

0 

- 012/1 

— 012/2 

- 013/2 

- 013/3 

A + 1 

0 

- 013/1 

— 04/1 

— 04/2 

- 014/3 

A + 1 

0 

0 

- 05/1 

- 014/2 

- 05/3 

A + 1 

























and this generates the characteristic equation of the system (fT8]l as follows, 


F(A) = (A + l)5-ai2345/| + /i(A,T) 

where, the function h{X, r) is given, 

= (A + 1) (A + 1)|( —a345/i/| — Q:35 /i/ 2(A + 1)| — Q:2/l|a34/|(A + 1) — a345/|/i 
+“ 345 / 1/3 + (A + l)^«4/2 + “ 2 / 2 ! — “ 345/3 “ “5(A + 1)^ — “35/2/3(A + 1) 

— “45/2/3(A + 1)| —“1/1 — “2/3I — (A + l)^“3/3 — “34/1/2/3 — “35/1 (A + 1)| 

— (A + 1)| — (A + 1)^Q;3/2 — “345/1/3I — “2 /i|“ 34/2 (A + 1)^ — “345/1 — “34/1/3I 
+“1/2! — “345/2/3 — “35 /i/2 (A + 1) + “345/1/3 } 

+“1/2 — “2/3|(A + 1)0:34/1 — 0345/1/! + 0345/^/3 + a4/2(A + 1)^1 

— (A + 1)|(A + 1)034/2/3 — 0345/f — “34 /i/3 (A + 1)| + O2/2I — “345/2 


+“45/1 (A + 1 ) + “345/1/2/3 + “345/1/2/3I —“1/3 — “2/3I — “345/1 

— “ 5 /i(A + 1 )^ — 035 / 2 / 3 (A + 1 ) — 045 / 2 / 3 (A + 1 )} “ (A + 1 )| “ “ 345 / 1/3 

— “35/2 (A + 1 ) — “ 45 /i/ 3 (A + 1 )| + 02/l| — “345/2 + “45/1 (A + 1 ) 
+“345/1/2/3 + “345/1/2/3I 


(19) 


„—XT\n 


Thus, in this model also, the characteristic equation contains the terms (A + 1)"", (1 — f5e~ 
associated with the product of all o/s, and a function of A and r. The inhibitory influence /3 
represents the one measure during delay in the loop containing n neurons. 


Generalization of the DNN Model 

We now generalized the dynamic neural network model by extending the state space to N 
dimensions. We introduce the network of N neurons with cross-linking among them, and the 
local feedback responses are made in cyclic order of j + 2 neurons, where j = 1,2, 3,..., N — 2. 
The synaptic delay of the neurons differs from each other, nevertheless, we consider the time 
delays to be identical in the analysis(rj = r). We assume that the pathways within a cyclic 
network of neurons has the same measure of the inhibitory influences of the past history, f3j 
over the loop, however, the measures remain varying over each and every cyclic network of j + 2 
neurons. Thus, the Wneuron model is described by. 


N-2 


Xi{t) = -Xi{t) + “i ^ tanh Xi+j+i{t) - l5jXi+j+i{t - r) 


( 20 ) 
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where i(mod N) and the term symbols have their usual meanings. Similarly, using linearization 
process around the origin, DDE (I20p can be written as 


n—2 

J=1 


( 21 ) 


The forms of the solutions are similar to those deployed in the previous sections. The Jacobian 
of (j21l) has the following characteristics: 

(i) For lower triangular region of the matrix, each entry of the diagonal having (n — j) elements 
have the following form all the way alongside the main diagonal: 


—Q;j+i(l — e where j < i < N 

0 


1, for each 1 < j < — 2 

, for j = — 1 


(a) For upper triangular region of the matrix, diagonal having {n — j) elements have the entries 
alongside the main diagonal in the form of 


-aAr_fc(l -/3j_i e 
0 


where N 


1 < ^ < J, for each 2 < j < A^ — 1 
, for j = 1 


and 

{in) For j = 0, i.e. diagonal elements, each entry is of the form (A + 1) where A is the eigenvalue. 

Theorem N.l: The general characteristic equation for the linear form of the system of DNN 
exists in the form of 

N 

F{\) = (A + 1)^ + (-1)^(1 - n «* + (22) 

where /? represents the measure of inhibitory influence during tn delay in the largest pathway 
containing all the N neurons. The function y(A, r) has to be determined from the expansion of 
the Jacobian of N dimention. 

1 Discussions and Conclusions 

Four new models of DNN, one each in 1-D, 4-D, 5-D and A^-D, have been developed in this small 
piece of work. Our models(l-D, 4-D, 5-D and n-D) are the transformation and extensions of the 
existing models in In each model, we perform the local stability analysis and consequently, 
we are able to determine the zero root of the characteristic equation of the linear form of each 
and every model equation, excluding the generalised model. The zero root has multiplicity 2 
in 2-D, 3-D and 4-D under certain conditions. In 1-D model, the zero eigenvalue is a single 
root when the feedback strength is one unit. The generalised n-D model is an extension from 
the smaller 3-D, 4-D and 5-D models. From the analysis in these models, we can conclude that 
there always exist certain conditions, in every model of dimension n > 1, for the existence of the 
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zero root and the subsistence of multiplicity 2. Moreover, we can observe a similar pattern in 
the characteristic equations of all the models, although one is different from each other owing to 
the forms of the equations. The general characteristic equation of DNN is very complicated to 
obtain as it is quite cumbersome to evaluate the value of higher order determinants analytically 
and there has not been a determinate form of the equation, although there are general formula 
like Leibnitz’s rule, Laplace’s formula, etc. As a result, we derive the general characteristic 
equation (I22jl with some indeterminate terms expressed as function(s) of A and r and show the 
similar terms involved. These findings may yield innovative results on the dynamics of neural 
networks and provide better insights into the mathematical models of DNNs. 
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